Systems and methods for using ai to identify regions of interest in medical images

ABSTRACT

Examples of the present disclosure describe systems and methods for using AI to identify regions of interest (ROI) in medical images. In aspects, medical reports and images may be provided to a first environment. The first environment may use the medical report data/medical images to train a natural language processing (NLP)-based algorithm to identify the location in images of ROI described in the medical report data. The output of the NLP-based algorithm may be stored in an ROI repository in the first environment. After the NLP-based algorithm has been trained, a request to train a user-specific model may be received in a second environment. Data objects for the requested user-specific model may be provided to the first environment, which uses the ROI repository to train the model. The trained model may be provided to the second environment, where the trained user-specific model/algorithm may be tested and stored.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of U.S. ProvisionalApplication No. 63/116,162, filed on Nov. 20, 2020, entitled “Systemsand Methods for Using AI to Identify Regions of Interest in MedicalImages,” the disclosure of which is hereby incorporated by reference inits entirety.

BACKGROUND

In the realm of artificial intelligence (AI), deep learning enablessystems to automatically discover the information required to performfeature detection or classification using raw data. Deep learningrequires access to large amounts of accurately labeled data. Typically,the data labeling is primarily a manual process, which can beprohibitively costly in terms of time and human/financial resources.Moreover, data privacy concerns are often a consideration.

It is with respect to these and other general considerations that theaspects disclosed herein have been made. Also, although relativelyspecific problems may be discussed, it should be understood that theexamples should not be limited to solving the specific problemsidentified in the background or elsewhere in the present disclosure.

SUMMARY

Examples of the present disclosure describe systems and methods forusing AI to identify regions of interest (ROI) in medical images. Inaspects, medical report data and/or corresponding medical images may beprovided to a first service or application in a first environment. Thefirst service/application may use the medical report data/medical imagesto train a natural language processing (NLP)-based algorithm to identifywithin the medical images the location of findings described in themedical report data. The output of the NLP-based algorithm may be storedin an ROI repository in the first environment. After the NLP-basedalgorithm has been trained, a request to train a user-specific model oran algorithm may be received by a second service or application in asecond environment. In response to the request, one or more data objectsfor the requested user-specific model/algorithm may be provided to thefirst service/application in the first environment. The firstservice/application may use data in the ROI repository to populate thedata objects and train the user-specific model/algorithm. The traineduser-specific model/algorithm may then be provided to the secondservice/application in the second environment, where the traineduser-specific model/algorithm may be tested, stored, and/or provided tothe user.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Additionalaspects, features, and/or advantages of examples will be set forth inpart in the description which follows and, in part, will be apparentfrom the description, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive examples are described with reference tothe following figures.

FIG. 1 illustrates an overview of an example system for using AI toidentify ROI in medical images, as described herein.

FIG. 2 is a diagram of an example process flow for using AI to identifyROI in medical images, as described herein.

FIG. 3 illustrates an example method for training an NLP-based model togenerate medical images comprising labeled ROI, as described herein

FIG. 4 illustrates an example method for using AI to identify ROI inmedical images, as described herein.

FIG. 5 illustrates one example of a suitable operating environment inwhich one or more of the present embodiments may be implemented.

DETAILED DESCRIPTION

Medical imaging has become a widely used tool for identifying anddiagnosing abnormalities, such as cancers or other conditions, withinthe human body. Medical imaging processes such as mammography andtomosynthesis are particularly useful tools for imaging breasts toscreen for, or diagnose, cancer or other lesions with the breasts.Tomosynthesis systems are mammography systems that allow high resolutionbreast imaging based on limited angle tomosynthesis. Tomosynthesis,generally, produces a plurality of X-ray images, each of discrete layersor slices of the breast, through the entire thickness thereof. Incontrast to conventional two-dimensional (2D) mammography systems, atomosynthesis system acquires a series of X-ray projection images, eachprojection image obtained at a different angular displacement as theX-ray source moves along a path, such as a circular arc, over thebreast. In contrast to conventional computed tomography (CT),tomosynthesis is typically based on projection images obtained atlimited angular displacements of the X-ray source around the breast.Tomosynthesis reduces or eliminates the problems caused by tissueoverlap and structure noise present in 2D mammography imaging.

In recent times, artificial intelligence (AI) has been increasingly usedto evaluate the image data generated using medical imaging. Inparticular, machine learning methods, such as deep learning, providepowerful tools for evaluating image data. Although such tools are highlyaccurate and efficient, these tools must be trained to perform specifictasks. The training requires access to a large amount of accuratelylabeled data. Generally, the data labeling process is performedmanually. For example, a clinical professional must read medical reportdocuments (physician notes, radiology reports, biopsy reports, etc.) toidentify ROI associated with a patient. The clinical professional labelsthe identified ROI on medical images associated with the medicaldocuments. Often, the quality of the labeling varies among clinicalprofessionals based on various factors, such as experience, ability,fatigue, etc. The labeled medical images are provided as input to an AIcomponent. Based on the input, the AI component is trained to identifythe labeled ROI in medical images subsequently provided to the trainedAI component. When the clinical professional intends to train the AIcomponent to identify a new ROI or a new aspect of a ROI for which theAI component was previously trained, the entire process must berepeated. Thus, the data labeling process is often time-consuming,cumbersome, expensive, and potentially inaccurate.

In addition, the large amount of accurately labeled data includespatient records and other sensitive physical information that isprotected by various laws and regulations including data security andprotection and confidential handling of protected health information. Assuch, to comply with the laws and regulations, the data for purposes oflabeling must first be deidentified by removing identification of aparticular patient prior to the export of data from a medical facility.Such deidentification is time consuming and often done manually.

To address such issues with data labeling for AI training, the presentdisclosure describes systems and methods for using AI to identify ROI inmedical images. In aspects, a first computing environment may comprisesensitive physical and/or electronic data, such as the medical reportdata, medical images, patient records, and other hospital informationsystem (HIS) data. The first computing environment may correspond to ahealthcare facility or a section or deport intent of a healthcarefacility. At least a portion of the medical report data and/or medicalimages may be provided as input to a first service or application in thefirst computing environment. The first service or application may usethe input to train an AI model or algorithm to identify ROI within themedical images based on the medical report data. In at least oneexample, the model or algorithm may use NLP techniques to identifylanguage that describes the locations of findings in the medical reportdata. The model or algorithm may use the identified language to provideoutput including image overlays for the medical images or annotatedversions of the medical images that include labeled locations of thefindings identified by the identified language. The labeled locationsmay include textual labels, numerical values, highlighting, encircling(and/or other types of content enclosing), arrows or pointers, font orstyle modifications, etc. The output of the model or algorithm may bestored in at least one data repository in the first computingenvironment. The data repository may also store one or more portions ofthe medical report data and/or the patient records.

In aspects, a second computing environment may include a second serviceor application for training and storing user-requested models oralgorithms. The second computing environment may be physically and/orlogically separate from the first computing environment. In response toreceiving a request to train a user-requested model or algorithm, thesecond service or application may provide data objects and/or trainingrequirements for the requested user-specific model or algorithm to atraining component in the first computing environment. The trainingcomponent may search the data repository to identify informationrelevant to the requested user-specific model or algorithm. The trainingcomponent may use the identified information to train the requesteduser-specific model or algorithm. The trained user-specific model oralgorithm may be provided to the second service or application in thesecond computing environment without allowing the second computingenvironment access to the sensitive data in the first computingenvironment. Thus, the integrity and security of the sensitive data maybe maintained throughout the training process. Upon receiving thetrained user-specific model or algorithm, the second service orapplication may evaluate the model to determine a set of performancemetrics. The set of performance metrics may represent the accuracy oreffectiveness of the trained user-specific model or algorithm. In atleast one aspect, the second service or application may use the set ofperformance metrics to iteratively tune/train the trained user-specificmodel or algorithm.

Accordingly, the present disclosure provides a plurality of technicalbenefits including but not limited to: training an NLP-based model todetect text relating to ROI locations, using NLP-based model output totrain specific AI models, improving data security/privacy during modelcreation, improving the accuracy of labeled data, improving theefficiency of generating labeled data, enabling self-learning AI systemswithin client or sensitive environments.

FIG. 1 illustrates an overview of an example system for using AI toidentify regions of interest (ROI) in medical images as describedherein. Example system 100 as presented is a combination ofinterdependent components that interact to form an integrated system.Components of system 100 may be hardware components or softwarecomponents implemented on and/or executed by hardware components of thesystem. System 100 may provide one or more operating environments forsoftware components to execute according to operating constraints,resources, and facilities of system 100. In one example, the operatingenvironment(s) and/or software components may be provided by a singleprocessing device, as depicted in FIG. 6. In another example, theoperating environment(s) and software components may be distributedacross multiple devices. For instance, input may be entered on a userdevice and information may be processed or accessed using other devicesin a network, such as one or more network devices and/or server devices.

As one example, system 100 comprises environments 101 and 121 andnetwork 110. One of skill in the art will appreciate that the scale ofsystems such as system 100 may vary and may include more or fewerenvironments and/or components than those described in FIG. 1. Forinstance, in some examples, at least a portion of the functionality andcomponents of environments 101 and 121 may be integrated into a singleenvironment, processing system, or device. Alternately, thefunctionality and components of environments 101 and/or 121 may bedistributed across multiple environments or processing systems.

Environment 101 may comprise user devices 102A, 102B, and 102C(collectively “user devices 102”), server device 104, and data store(s)106. In at least one aspect, environment 101 may represent a cloud-basedor distributed computing environment. User devices 102 may be configuredto receive or collect input from one or more users or alternate devices.Examples of user devices 102 include, but are not limited to, personalcomputers (PCs), server devices, mobile devices (e.g., smartphones,tablets, laptops, personal digital assistants (PDAs)), and wearabledevices (e.g., smart watches, smart eyewear, fitness trackers, smartclothing, body-mounted devices). User devices 102 may include sensors,applications, and/or services for receiving or collecting input. Examplesensor include microphones, touch-based sensors, keyboards,pointing/selection tools, optical/magnetic scanners, accelerometers,magnetometers, gyroscopes, etc. The collected input may include, forexample, voice input, touch input, text-based input, gesture input,video input, and/or image input.

Server device 104 may be configured to receive collected input from userdevices 102. Examples of server device 104 include, but are not limitedto, application servers, web servers, file servers, database servers,and mail servers. Upon receiving collected input, server device 104 mayprovide access to data and one or more services/applications. The dataand services/applications may be stored remotely from server device 104and accessed by server device 104 via network 110. Alternately, the dataand services/applications may be stored and accessed locally on serverdevice 104 using a data store, such as data store(s) 106. Examples ofdata store(s) 106 include, but are not limited to, databases, filesystems, directories, flat files, and email storage systems. In someaspects, data store(s) 106 may comprise data objects and/or sets ofinstructions for one or more algorithms and/or models. A model, as usedherein, may refer to a predictive or statistical utility or program thatmay be used to determine a probability distribution over one or morecharacter sequences, classes, objects, result sets or events, and/or topredict a response value from one or more predictors. A model may bebased on, or incorporate, one or more rule sets, machine learning (ML),a neural network, or the like. In at least one aspect, the algorithmsand/or models may be proprietary and/or subject to trade secretprotections by the owners of the algorithms and/or models.

The algorithms and/or models in data store(s) 106 may be used to performone or more specific tasks, such as identifying a type of cancer, acategory of disease, image anomalies, etc. Although reference tospecific types of tasks are described herein, it is contemplated thatthe algorithms and/or models may be used to perform alternate types oftasks and used in alternate types of environments. In response toreceiving the collected input, server device 104 may collect or receiveone or more data objects and/or sets of instructions relating to aspecific task or set of tasks from data store(s) 106. Server device 104may identify a task and/or corresponding data objects/instructions basedon one or more terms in or associated with the collected input. Forexample, server device 104 may parse the collected input to identifyquery terms or input terms. The identified terms may be used to searchthe data (e.g., algorithm names, data object text, instruction text) indata store(s) 106 for similar or matching terms using search techniques,such as pattern matching, regular expressions, fuzzy matching, etc. Whenone or more matches are identified, the correspondingalgorithm(s)/model(s) may be selected and server device 104 may collector receive one or more data objects and/or sets of instructions relatingto the selected algorithm(s)/model(s). Server device 104 may provide oneor more data objects and/or sets of instructions to environment 121based on the collected input.

Server device 104 may be further configured to evaluate response datareceived from environment 121. The response data may be provided byenvironment 121 in response to one or more data objects and/or sets ofinstructions provided to environment 121. In aspects, server device 104may comprise or provide access to an execution environment (notpictured). The execution environment may comprise or utilizefunctionality for evaluating the response data. In at least one aspect,the response data corresponds to a trained user-requested model oralgorithm. The evaluated response data may be stored in one or more datastores, such as data store(s) 106. The response data may be provided toa user in response to receiving the collected input.

Environment 121 may comprise server device 124, data store(s) 126, andfeature store(s) 128. In at least one aspect, environment 121 mayrepresent a computing environment comprising sensitive data, such as ahealthcare computing environment comprising patient data. Server device124 may be configured to collect data from the one or more data sources,such as data store(s) 126 and/or feature store(s) 128. Examples of datastore(s) 126 and feature store(s) 128 include, but are not limited to,databases, file systems, directories, flat files, and email storagesystems. In at least one aspect, the collected data may correspond tomedical report data, medical images, patient records, and/or othersensitive medically related information. The collected data may be usedto train an NLP-based algorithm or model (not pictured). At least aportion of the output of the trained NLP-based algorithm or model and/orthe collected data may be stored in feature store(s) 128.

Server device 124 may be further configured to receive one or more dataobjects and/or sets of instructions from environment 101. Server device124 may identify a specific task associated with the received one ormore data objects and/or sets of instructions. The identified specifictask may be used to search feature store(s) 128 for stored data relevantto performing the specific task. In at least one aspect, the stored datamay correspond to labeled or annotated image data, text terms or phrasesfrom medical report data, and/or feature data associated with image dataor medical report data. Stored data identified to be relevant may beprovided to a training component (not pictured) within environment 121.The training component may be a hardware device, a software componentwithin server device 124, or a software component within a separatehardware device of environment 121. In examples, the training componentmay be implemented as a black box that provides separation betweenenvironment 101 and environment 121. The separation may preventenvironment 101 (and other environments external to environment 121)from accessing the sensitive data of environment 121 from outside ofenvironment 121. The separation may also prevent environment 121 fromunauthorized access of the models and/or algorithms stored in datastore(s) 106. For instance, as the models and/or algorithms may beproprietary to owners who are third parties with respect to environment101, it may be desirable for the owners to keep the algorithms securefrom users in environment 101.

The training component may be configured to train a user-requested modelor algorithm. In examples, the stored data identified to be relevant maybe provided to the training component. The training component may usethe relevant stored data to train a user-requested model or algorithmthat is operable to perform the identified specific task. The traineduser-requested model or algorithm may then be provided as response datato environment 101. In aspects, the user-requested model or algorithmmay be trained and provided to environment 101 such that sensitive datain environment 121 is not exposed to environment 101. As such, thepatient data that is located in any sensitive medically relatedinformation used to train the trained user-requested model or algorithmdoes not need to be de-identified because it is not removed from theenvironment 101 and stays on site in the environment 101. This resultsin saving significant time from gathering, processing, exporting andstoring information, which previously may have been done manually by ahighly skilled medical technician.

FIG. 2 is a diagram of an example process flow for using AI to identifyregions of interest (ROI) in medical images, as described herein.Process flow 200, as presented, comprises environments 201 and 221. Inexamples, environment 201 may represent a healthcare facility, such as ahospital, an imaging and radiology center, an urgent care facility, amedical clinic or medical offices, an outpatient surgical facility, aphysical rehabilitation center, etc. Environment 201 may comprisesensitive or private information associated with a healthcare facility,healthcare patients, and/or healthcare personnel. Environment 221 mayrepresent a web-based, cloud-based, or distributed computing environmentassociated with environment 201. Environment 221 may be publicly orselectively accessible and may implement security procedures to enablethe secure access of environment 201. Generally, environment 221 may notstore or have access to sensitive or private information comprised byenvironment 201. Environments 201 and 221 may be physically and/orlogically separated. In addition, the environment 201 and 221 may beseparated by firewalls and authentication protocols ensuring safehandling of the sensitive medical information comprised in theenvironment 201.

Environment 201 may comprise ROI analysis engine 202, medical data 204A,204B, and 204C (collectively “medical data 204”), ROI repository 206,orchestration engine 214, and training engine 216. Environment 221 maycomprise user(s) 208, application 210, algorithm repository 212, andmodel repository 218. One of skill in the art will appreciate that thenumber and type of environments and/or components associated withenvironment 201, environment 221, and process flow 200 may vary fromthose described in FIG. 2.

In aspects, ROI analysis engine 202 may be provided, or may have accessto, medical data associated with one or more patients, such as medicaldata 204. ROI analysis engine 202 may be configured to identify ROIassociated with medical data 204. Examples of medical data 204 include,but are not limited to, medical report data 204A (e.g., radiologyreports, biopsy reports, audio reports, healthcare professional notesand documents), medical image data 204B (e.g., X-ray images, CT images,MRI images, ultrasound images), and electronic medical record (EMR) data204C (e.g., patient records, medical and treatment history information,patient health data). Although specific references to medical data andprocedures are described, it is contemplated that the systems andmethods described herein may be implemented with alternate types ofdata, procedures, and environments.

Upon receiving the medical data, ROI analysis engine 202 may use medicaldata 204 to train an AI model/algorithm (not pictured) withinenvironment 201. The AI model/algorithm may be stored by ROI analysisengine 202 or elsewhere within environment 201. The AI model/algorithmmay be configured to identify ROI within the medical image data based oncorresponding medical report data. For example, the AI model/algorithmmay implement NLP techniques to identify text and/or speech in medicalreport data that describes the locations of one or more findings withinthe patient. The AI model/algorithm may use the identified text and/orspeech to identify the findings in corresponding medical image data. TheAI model/algorithm may label the identified finding within the medicalimage data by generating image overlays or annotated versions of themedical image data. The medical image data labeled by the AImodel/algorithm, image feature data relating to the medical image data,and the corresponding identified text and/or speech may be stored in adata store, such as ROI repository 206.

After the AI model/algorithm had been trained, a user in or interfacingwith environment 221, such as user(s) 208, may access application 210.Examples of user(s) 208 may include one or more manufacturers ofalgorithms designed to detect different types of medical conditions orabnormalities, such as cancers which may be diagnosed by healthcareprofessionals from medical images. Application 210 may be configured toreceive, store, and/or process user requests to train a user-specificalgorithm to perform a specific task. Upon receiving a request fromuser(s) 208 to train a new user-specific algorithm, application 210 mayaccess algorithm repository 212. Algorithm repository 212 may beconfigured to store and provide various algorithms relating toenvironment 201. The algorithms of algorithm repository 212 may relateto various topics, concepts, or areas. For example, a first algorithmmay be used to identify a first type of cancer, a second algorithm maybe used to identify a second type of cancer, and a third algorithm maybe used to identify images having poor image quality. Algorithmrepository 212 may be configured to store and provide data objectsand/or instructions for training the stored algorithms. Algorithms inthe algorithm repository 212 may be proprietary and subject to tradesecret protections. It may be desirable for the owners of the algorithmsto keep the algorithms secure. As discussed above, environments 221 and201 may be physically and logically separated and protected by firewallsand other security measures. By separating the environments 221 and 201access to the algorithms is secured and can be managed by the owners asthey reside in the environments subject to the owners' control.

Application 210 may use terms and keywords in the request from user(s)208 to identify a context (e.g., a topic, a concept, or an area)associated with the request. Application 210 may use the identifiedcontext to search algorithm repository 212 for relevant algorithms. Whena relevant algorithm is identified in algorithm repository 212, theidentified algorithm, one or more data objects, and/or instructions fortraining the identified algorithm may be provided to orchestrationengine 214. In some examples, orchestration engine 214 may be configuredto monitor environment 221 and/or application 210 to detect when a userrequest to train a user-specific algorithm is received by application210. The monitoring may include the implementation of monitoringservices or software used to transmit periodic queries to application210, receive notifications from application 210, intercept messagesbetween users(s) 208 and application 210, etc. When a user request totrain a user-specific algorithm is detected, orchestration engine 214may cause algorithm repository 212 to provide the identified algorithm,one or more data objects, and/or instructions for training theidentified algorithm to orchestration engine 214 and/or training engine216. For example, orchestration engine 214 may request the access pathand/or credentials for algorithm repository 212. Orchestration engine214 may use the access path and/or credentials to retrieve theidentified algorithm, data objects, and/or instructions. Alternately,orchestration engine 214 may provide the access path and/or credentialsto training engine 216 and training engine 216 may use the access pathand/or credentials to retrieve the identified algorithm, data objects,and/or instructions.

Orchestration engine 214 and/or training engine 216 may also beconfigured to initiate the training of the identified algorithm withinenvironment 201. Orchestration engine 214 may provide the identifiedalgorithm, one or more data objects, and/or instructions for trainingthe identified algorithm to training engine 216. Alternately oradditionally, orchestration engine 214 may provide a command (includingparameters) for initiating the training of the identified algorithm tothe training engine 216. Training engine 216 may be configured to searchROI repository 206 for data (e.g., medical image data, image featuredata, identified text and/or speech) associated with the identifiedalgorithm, and to train a model based on the data. In aspects, trainingengine 216 may be implemented in a manner that provides separationbetween environment 201 and environment 221. For example, trainingengine 216 may prevent users and devices in environment 221 (and otherenvironments external to environment 201) from accessing the sensitiveor secure data of environment 201, such as medical data 204, fromoutside of environment 201. Further, training engine 216 may preventusers and devices in environment 201 (and other environments external toenvironment 221) from directly accessing the algorithms stored inalgorithm repository 212. For instance, training engine 216 mayimplement security features or policies that prevent users and devicesin environment 201 and environment 221 from viewing or accessing thedata (e.g., ROI repository 206 data or algorithm repository 212)received by training engine 216.

Upon receiving the identified algorithm, one or more data objects,instructions for training the identified algorithm, and/or command(including parameters) for initiating the training of the identifiedalgorithm, training engine 216 may train a model based on the identifiedalgorithm. When the model has been trained, orchestration engine 214 ortraining engine 216 may provide the trained model to model repository218. Alternately, orchestration engine 214 or training engine 216 mayprovide the trained model to application 210 and application 210 mayprovide the trained model to model repository 218. Model repository 218may be configured to store various trained models and associated data,such as creation/modification data, a description of the model, testingdata, result accuracy data, keywords or terms associated with the model,version/iteration number, etc.

In aspects, after the trained model has been provided to modelrepository 218 and/or application 210, user(s) 208 may interact with thetrained model using application 210. For example, application 210 mayalso be configured to provide a testing environment (not pictured) totest the trained model. The testing environment may implement tools forevaluating the performance metrics for the trained model. In examples,the performance metrics may relate to receiver operating characteristics(ROCs) and/or free-response receiver operating characteristics (FROCs),such as sensitivity, specificity, precision, hit rate, accuracy, etc.Evaluating the performance metrics for the trained model may includeusing the trained model to perform a specific task intended by the userand/or comparing the performance metrics for the trained model to a setof baseline performance metrics. For example, the trained model may beused to identify image data or aspects thereof. Based on the performancemetrics for the trained model, the trained model may be provided totraining engine 216, as described above, to be refined/retrained. A setof training parameters may for refining/retraining may also be providedto training engine 216. Training engine 216 may refine/retrain thetrained model based on the set of training parameters. Therefined/retrained model may be provided to application 210 and/or modelrepository 218. The testing environment of application 210 may be usedto evaluate the performance metrics of the refined/retrained model. Insome aspects, the performance metrics of trained model and therefined/retrained model may be compared to determine whether the trainedmodel or the refined/retrained model is more accurate. Based on thecomparison, the trained model and/or the refined/retrained model may bestored or removed from the model repository 218. Additionally, therefined/retrained model may be further refined/retrained using theprocess described above.

Having described a system and process flow that may employ thetechniques disclosed herein, the present disclosure will now describeone or more methods that may be performed by various aspects of thepresent disclosure. In aspects, methods 300 and 400 may be executed by asystem, such as system 100 of FIG. 1. However, methods 300 and 400 arenot limited to such examples. In other aspects, methods 300 and 400 maybe performed by a single device comprising multiple computingenvironments. In at least one aspect, methods 300 and 400 may beexecuted (e.g., computer-implemented operations) by one or morecomponents of a distributed network, such as a web service/distributednetwork service (e.g. cloud service).

FIG. 3 illustrates an example method for training an NLP-based model togenerate images comprising labeled ROI, as described herein. Examplemethod 300 begins at operation 302, where medical data may be received.In aspects, an analysis component, such as ROI analysis engine 202, maybe located in a secure environment, such as environment 201. The secureenvironment may correspond to a client environment of a healthcarefacility or of another location comprising sensitive data. The analysiscomponent may receive or have access to medical data from one or moresources, such as data store(s) 106. The medical data may include medicalreport data, medical image data, EMR data, and other HIS data.

At operation 304, text describing the location of ROI may be identified.In aspects, the analysis component may apply one or more NLP techniquesto the medical data. Example NLP techniques include, but are not limitedto, named entity recognition, sentiment analysis, tokenization, sentencesegmentation, and stemming and lemmatization. The NLP techniques may beused to identify significant terms and/or phrases in text data of themedical data. The significant terms and/or phrases may correspond toterms and/or phrases of a standardized (or semi-standardized) lexiconused for reporting the outcomes of image review. As one example, the NLPtechniques may be applied to medical report data (e.g., radiologyreports and/or biopsy reports) to identify text describing one or morefindings or ROI (e.g., lesions, asymmetric breast tissue,macrocalcifications, asymmetry density, distortion mass, or adenopathy)resulting from a mammographic exam. The text may include features of thefindings or ROI, such as size, location, texture, density, symmetry,etc. As a specific example, the NLP techniques may identify a sentencein a radiology report that indicates a lesion was detected in thesuperior medial portion of a patient's left breast. The NLP techniquesmay also identify another sentence in the radiology report thatindicates the size and density of the lesion and the approximatelocation of the lesion with in the superior medial quadrant. The textassociated with each sentence may be extracted by the analysiscomponent. The extracted text may be labeled (e.g., superior mediallesion) and stored with text relating to similar findings. For instance,all text describing findings or ROI in the superior medial quadrant of abreast may be stored under the category “Superior Medial Findings.”

At operation 306, an NLP-based model may be trained. In aspects, thesignificant terms and/or phrases identified in the text data of themedical data (and in other medical data) may be provided as input to anNLP-based model located within the secure environment. The NLP-basedmodel may be generated and/or maintained by the analysis component or byanother component within the secure environment. Image datacorresponding to the identified significant terms and/or phrases mayalso be provided as input to an NLP-based model. The input may be usedto train the NLP-based model to match the identified significant termsand/or phrases to corresponding locations of ROI in the image data.Matching the identified significant terms and/or phrases to thecorresponding locations may include generating labeled image datacomprising labels and/or annotations of the ROI. For example, varioustext strings from a radiology report and one or more correspondingtomosynthesis computer-aided design (CAD) images may be provided to anNLP-based model. In response to the text string “a lesion was detectedin the superior medial portion of a patient's left breast,” theNLP-based model may evaluate the CAD image(s) to identify images of thepatient's left breast. For each identified CAD image of the patient'sleft breast, the NLP-based model may evaluate the superior medialquadrant of the breast in the CAD image to identify ROI corresponding tothe text string. The evaluation may include the use of unified vectorsof location features. For each identified ROI, the NLP-based model maylabel the ROI on the CAD image and/or create a labeled version of theCAD image. For instance, the NLP-based model may generate an overlay inwhich the identified ROI is encircled or otherwise highlighted.

At operation 308, the NLP-based model data may be stored in arepository. In aspects, content generated or output by the NLP-basedmodel may be stored in a data repository, such as ROI repository 206.The content may include, labeled or otherwise annotated image data,unlabeled/unannotated image data, feature vectors, terms and/or phrases,and/or medical data available to the analysis component. The datarepository may also be located in the secure environment such that theNLP-based model may be trained, and the content generated therefrom maybe stored without exposing sensitive data to entities outside of thesecure environment.

FIG. 4 illustrates an example method for using AI to identify ROI inmedical images as described herein. In aspects, an application orservice providing component (“application component”), such asapplication 210 may be located in a first computing environment, such asenvironment 221. The first computing environment may correspond to acloud-based or web-based environment that may be publicly or selectivelyaccessible. The first computing environment may provide access to asecond computing environment, such as environment 201. The secondcomputing environment may correspond to a secure, client environment ofa healthcare facility or of another location comprising private orsensitive data.

Example method 400 begins at operation 402, where a request to train auser-selected algorithm is detected. In aspects, a user in or accessingthe first computing environment may access a user interface provided bythe application component. The user interface may provide the user withan option to identify an algorithm to be trained to perform one or moretasks. Identifying the algorithm may comprise selecting an algorithmfrom a list of algorithms in an algorithm store, such as algorithmrepository 212. Alternately, identifying an algorithm may compriseproviding one or more algorithm characteristics (e.g., intended functionor type/category) to the user interface. In response to a useridentifying an algorithm, the application component may provide theidentified algorithm, one or more data objects associated with theidentified algorithm, and/or instructions for training the identifiedalgorithm (collectively, “algorithm container”).

At operation 404, the algorithm container may be provided to the secondcomputing environment. In some aspects, the application component maysend the algorithm container to one or more components in the secondcomputing environment in response to receiving the user request. Forexample, the application component may send the algorithm container toan algorithm training orchestration component, such as orchestrationengine 214, or to an algorithm training component, such as trainingengine 216. In other aspects, the algorithm training orchestrationcomponent of the second computing environment may monitor theapplication component in the first computing environment. Upon detectinga request to train a user-selected algorithm has been received by theapplication component, the orchestration component or the algorithmtraining component may request the algorithm container from theapplication component or the algorithm store. In response to the requestby the orchestration component, the application component may providethe algorithm container to the orchestration component or the algorithmtraining component. Alternately, the application component may provideinformation for the algorithm container (e.g., identifier,location/path, access credentials) to the orchestration component oralgorithm training component. The orchestration component or algorithmtraining component may use the information for the algorithm containerto retrieve the algorithm container.

At operation 406, the algorithm container may be used to identifycontent related to the user-selected algorithm. In aspects, one or moreidentifiers (e.g., terms, phrases, topics, contexts) associated with thereceived algorithm container may be identified. The identifiers may beused to search a data repository, such as ROI repository 206, forcontent related (e.g., relevant) to the algorithm container. The contentin the data repository may include, for example, labeled or otherwiseannotated image data, unlabeled/unannotated image data, ROI featurevectors, terms and/or phrases describing ROI, and/or other medical dataavailable in the second computing environment. Searching the datarepository may include using pattern matching techniques, such asregular expressions, fuzzy logic, pattern recognition models, etc. Anycontent determined to be related to the algorithm container may beidentified and extracted from the data repository. As on specificexample, an algorithm container for detecting metastatic breast cancermay be titled as or comprise the term “Metastatic.” Based on identifyingthe term “metastatic” in/for the algorithm container, image datacomprising ROI that include instances of metastatic breast cancer may beidentified.

At operation 408, content related to the algorithm container may be usedto train a model. In aspects, the algorithm container and/or the contentrelated to the algorithm container may be provided as input to thetraining component in the second computing environment. The trainingcomponent may use the input to train a model corresponding to thealgorithm container. For example, the training component may use overlayimage data in the related content to populate or otherwise configure oneor more data objects in the algorithm container according to a set ofinstructions and/or parameters in the algorithm container. Thepopulated/configured data objects may be used to construct a modelrepresenting the algorithm the user requested to be trained. Inexamples, the model may be trained such that data used to train themodel in the second computing environment is not exposed to the firstcomputing environment.

At operation 410, the trained model may be provided to the firstcomputing environment. In aspects, the orchestration component mayreceive or collect the trained model from the training component. Theorchestration component may provide the trained model to the firstcomputing environment. For example, the orchestration component mayprovide the trained model to the application component and/or to a modelstore of the first computing environment, such as model repository 218.Alternately, the training component may provide the trained model to thefirst computing environment. The trained model may be stored in themodel store and/or presented to the user using the user interface. Theuser interface may enable the user to execute, modify, or otherwiseinteract with the trained model.

At operation 412, the trained model may be evaluated. In aspects, thefirst computing environment or a component thereof, such as theapplication component, may comprise a test operating environment. Thetest operating environment may provide one or more tools for evaluatingthe trained model. The evaluation may include identifying performancemetrics for the trained model and/or comparing the identifiedperformance metrics to a set of baseline or default performance metrics.In some aspects, the test operating environment may enable the iterativetraining of a model. For example, after evaluating a trained model inthe test operating environment, an updated algorithm container may bemanually or automatically selected from the algorithm store or mayotherwise be acquired. The updated algorithm container may be selectedby, for example, the application component based on predefined testingconstraints or according to a test script or executable test file forthe selected algorithm or algorithm type. The trained model and theupdated algorithm container may be provided to the training component inthe second computing environment. The updated algorithm container maycomprise an updated set of instructions and/or parameters for trainingthe trained model. Based on the updated algorithm container, thetraining component may update/(re)train the trained model. The updatedtrained model may be provided to the first computing environment. Thetest operating environment may be used to evaluate performance metricsfor the updated trained model. The performance metrics for the trainedmodel and the performance metrics for the updated trained model may thenbe compared to determine the which model (e.g., trained model or updatedtrained model) is more accurate. Based on the comparison, the mostaccurate model may be selected, and a newly updated algorithm containermay be selected or obtained. The process may continue as described aboveuntil a set of performance metrics meeting or exceeding a thresholdvalue/level is acquired, or until a defined set of criteria is met.

FIG. 5 illustrates an exemplary suitable operating environment for theautomating clinical workflow decision techniques described in FIG. 1. Inits most basic configuration, operating environment 500 typicallyincludes at least one processing unit 502 and memory 504. Depending onthe exact configuration and type of computing device, memory 504(storing, instructions to perform the techniques disclosed herein) maybe volatile (such as RAM), non-volatile (such as ROM, flash memory,etc.), or some combination of the two. This most basic configuration isillustrated in FIG. 5 by dashed line 506. Further, environment 500 mayalso include storage devices (removable, 508, and/or non-removable, 510)including, but not limited to, magnetic or optical disks or tape.Similarly, environment 500 may also have input device(s) 514 such askeyboard, mouse, pen, voice input, etc. and/or output device(s) 516 suchas a display, speakers, printer, etc. Also included in the environmentmay be one or more communication connections 512, such as LAN, WAN,point to point, etc. In embodiments, the connections may be operable tofacility point-to-point communications, connection-orientedcommunications, connectionless communications, etc.

Operating environment 500 typically includes at least some form ofcomputer readable media. Computer readable media can be any availablemedia that can be accessed by processing unit 502 or other devicescomprising the operating environment. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other non-transitory medium whichcan be used to store the desired information. Computer storage mediadoes not include communication media.

Communication media embodies computer readable instructions, datastructures, program modules, or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and includes anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared, microwave, and other wireless media.Combinations of the any of the above should also be included within thescope of computer readable media.

The operating environment 500 may be a single computer operating in anetworked environment using logical connections to one or more remotecomputers. The remote computer may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above as wellas others not so mentioned. The logical connections may include anymethod supported by available communications media. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets and the Internet.

The embodiments described herein may be employed using software,hardware, or a combination of software and hardware to implement andperform the systems and methods disclosed herein. Although specificdevices have been recited throughout the disclosure as performingspecific functions, one of skill in the art will appreciate that thesedevices are provided for illustrative purposes, and other devices may beemployed to perform the functionality disclosed herein without departingfrom the scope of the disclosure.

This disclosure describes some embodiments of the present technologywith reference to the accompanying drawings, in which only some of thepossible embodiments were shown. Other aspects may, however, be embodiedin many different forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments were provided sothat this disclosure was thorough and complete and fully conveyed thescope of the possible embodiments to those skilled in the art.

Although specific embodiments are described herein, the scope of thetechnology is not limited to those specific embodiments. One skilled inthe art will recognize other embodiments or improvements that are withinthe scope and spirit of the present technology. Therefore, the specificstructure, acts, or media are disclosed only as illustrativeembodiments. The scope of the technology is defined by the followingclaims and any equivalents therein.

What is claimed is:
 1. A method for using AI to identify ROI in medicalimages, the method comprising: detecting, by an orchestration engine ina first environment, a request to train an algorithm to identify theROI, wherein the request is provided to a second environment external tothe first environment; receiving an algorithm container for thealgorithm, wherein the algorithm container comprises one or more dataobjects for the algorithm; identifying, in a ROI repository in the firstenvironment, content related to the algorithm container, wherein thecontent comprises one or more images of the ROI; training, by a trainingengine in the first environment, a model using the content and providingthe trained model to the second environment.
 2. A system for using AI toidentify ROI in medical images, the system comprising: a processor; andmemory coupled to the processor, the memory comprising computerexecutable instructions that, when executed, perform a methodcomprising: detecting, by an orchestration engine in a firstenvironment, a request to train an algorithm to identify the ROI,wherein the request is provided to a second environment external to thefirst environment; receiving an algorithm container for the algorithm,wherein the algorithm container comprises one or more data objects forthe algorithm; identifying, in a ROI repository in the firstenvironment, content related to the algorithm container, wherein thecontent comprises one or more images of the ROI; training, by a trainingengine in the first environment, a model using the content and providingthe trained model to the second environment.
 3. A system for using AI toidentify ROI in medical images, the system comprising: a processor; andmemory coupled to the processor, the memory comprising computerexecutable instructions that, when executed, perform a methodcomprising: receiving, by an application in a first environment, arequest to train an algorithm to identify the ROI; providing, to asecond environment, an algorithm container for the algorithm, whereinthe algorithm container comprises one or more data objects for thealgorithm; receiving a trained model from the second environment,wherein the trained model is based on the algorithm container; andevaluating performance metrics of the trained model using a testoperating environment of the second environment.